AI is creating more work for data teams without corresponding budget increases, forcing engineers to maintain infrastructure instead of building. The article walks through architecture decisions and tool choices that let small teams scale operations without collapsing under demand.

• Only 30% of companies increased data budgets in 2025, yet expectations for governance, observability, and self-service features keep rising—most teams must deliver more with the same headcount
• Data volumes and citizen automations (via tools like n8n) multiply dependencies faster than teams can manage them; without proper orchestration and visibility, this creates invisible work that scales directly with data growth
• Pick infrastructure built for scale (Snowflake, Databricks) paired with a single control plane (like Orchestra) rather than a modular stack that requires constant manual intervention and cluster management

This newsletter covers major developments across data and AI, including DoorDash’s new Write-Ahead Intent Log for scalable CDC, OpenAI’s Kepler AI analyst for massive data insights, and Lyft’s Metric Semantic Layer for unified business metrics. It also reviews ClickHouse ingestion optimizations, DuckDB’s latest release and agent use cases, Databricks’ Lakehouse//RT and LTAP pushes, plus tools for self-healing pipelines and data quality monitoring.

• DoorDash replaced Debezium with a Write-Ahead Intent Log (WAIL) that writes mutation intents to Kafka and the DB, enabling faster recovery and much higher CDC throughput at Cassandra scale
• MotherDuck claims a $2.40/hour DuckDB instance beats a $64/hour Snowflake 2XL cluster by ~5x on ClickBench, with median query latencies of 3ms, positioning DuckDB for fleets of lightweight agent workloads
• Lyft's Metric Semantic Layer assigns dual business/operational ownership to "Golden Metrics" defined in versioned JSON, auto-propagating changes across dashboards, APIs, and AI agents to prevent divergent calculations
• OpenAI's Kepler analyst now operates over 600 PB of data using Codex-based crawlers and AST-normalized LLM grading to infer lineage, freshness, and hidden semantics automatically